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Abstract.  Managing  human  factors  in  nondestructive  evaluation  is  critical  to  maintain  inspection 
reliability.  Reliability  of  structural  health  monitoring  systems  is  particularly  sensitive  to  sensor 
degradation  over  time.  To  investigate  the  impact  of  these  issues,  probabilistic  models  for  risk  assessment 
and  cost-benefits  analysis  have  been  developed.  Quantitative  studies  are  presented  evaluating  the  effects 
of  variations  in  probability  of  detection  associated  with  human  factors,  plus  in-situ  sensor  degradation  on 
life  cycle  measures  such  as  cost  and  probability  of  failure. 

Keywords:  cost  benefit  assessment,  models,  nondestructive  evaluation,  probabilistic  risk  assessment, 
structural  health  monitoring 
PACS:  02.50.-r,  81.70.-q 

INTRODUCTION 

The  reliability  of  nondestructive  evaluation  techniques  is  critical  for  aircraft 
maintenance  programs.  Issues  in  nondestructive  inspection  discovered  through  probability 
of  detection  studies  has  generated  an  interest  in  determining  the  impact  of  inspection 
performance  on  total  service  life  [1],  For  inspection  problems  that  include  manual 
scanning,  complex  procedures,  and  low  frequencies  of  finding  critical  flaws,  there  is  a 
potential  for  some  critical  sites  to  not  be  inspected  effectively  due  to  the  requirements  of 
the  inspection  process,  or  there  are  inconsistent  requirements  for  calling  marginal  defects 
[2],  Options  to  address  NDE  reliability  associated  with  human  factors  include  improved 
NDE  design  of  equipment  and  procedures,  consistent  evaluation  of  NDE  reliability  through 
POD  studies,  and  improved  NDE  process  controls  using  quality  calibration  standards, 
increased  inspector  training  and  evaluation,  and  improved  management  oversight.  The 
application  of  structural  health  monitoring  (SHM)  system  with  in-situ  sensors  also  has  the 
potential  to  improve  inspection  reliability  by  eliminating  problematic  human  factors.  To 
properly  assess  the  best  approach  to  maintain  acceptable  inspection  reliability  and 
minimize  total  life  cost,  methods  to  evaluate  NDE  and  SHM  systems  using  probabilistic 
risk  assessment  with  cost-benefits  analysis  are  needed. 

The  application  of  in-situ  sensors  for  structural  health  monitoring  has  become  a 
important  research  topic  [3-5].  The  primary  benefit  of  structural  health  monitoring  (SHM) 
concerns  integration  with  prognostics,  where  the  management  of  high  value  assets  such  as 
military  aircraft  is  improved  through  the  quantitative  prediction  of  future  operating 
capability  and  accurate  determination  of  remaining  life.  However,  many  challenges  exist 
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for  the  practical  application  of  in-situ  sensors  for  SHM  including  1)  distinguishing  mission 
critical  defects  from  coherent  noise  features  present  in  distributed  sensor  signals,  2) 
sensitivity  to  dynamic  and  environmental  conditions,  3)  sensor  placement  for  optimal 
damage  state  observability,  and  4)  ensuring  the  reliability  of  the  on-board  SHM  system 
through  the  life  of  the  structure.  In  particular,  the  issue  of  degradation  of  the  in-situ  health 
monitoring  system  is  of  high  interest,  where  a  variety  of  sub-systems  such  as  sensors,  the 
bonds  between  sensors  and  structures,  the  wiring  harnesses,  the  measurement  hardware, 
and  the  power  (battery)  system  have  the  potential  to  decay  over  time. 

To  investigate  these  critical  issues  concerning  human  factors  in  NDE  and  in-situ  sensor 
degradation  in  SHM,  a  methodology  incorporating  cost  benefit  analysis  with  probabilistic 
risk  assessment  is  proposed.  Prior  work  has  addressed  development  of  a  strategy  and 
software  platform  to  enable  analysis  of  tradeoffs  in  NDE  and  SHM  design  in  terms  of 
product  life  cycle  outcomes  [6-7].  This  model  is  based  on  prior  work  by  Berens  et  al.,  who 
developed  a  software  tool,  PROF,  for  probabilistic  risk  assessment  of  fatigue  crack  growth 
and  fracture  incorporating  NDE  [8].  This  work  presents  the  development  of  probabilistic 
model  components  to  study  the  impact  human  factors  on  NDE  reliability  and  maintenance 
cost.  The  effects  of  variations  in  probability  of  detection  associated  with  human  factors  are 
explored.  Lastly,  probabilistic  models  incorporating  time-dependent  SHM  parameters  and 
case  studies  are  utilized  to  provide  insight  into  the  impact  of  SHM  sensor  degradation. 

HUMAN  FACTORS  IN  NONDESTRUCTIVE  EVALUATION 


There  are  a  variety  of  human  factors  that  impact  the  reliability  of  inspection  techniques 
and  potentially  degrade  performance  from  their  intrinsic  capability  of  detecting  defects  of 
varying  size  [9].  Several  studies  have  been  performed  to  understand  the  influence  of 
human  factors  on  NDE  performance  [1,10].  Key  human  factors  include  the  NDE  skill  level 
of  the  operator,  and  the  amount  of  operator  training  and  recent  experience  with  a  procedure 
and  equipment  of  interest.  The  psychological  state  of  the  operator  also  plays  a  critical  role 
in  inspection  performance  through  qualities  such  as  integrity,  concentration,  persistence, 
tolerance  to  the  environmental  conditions,  and  a  pre-existing  bias  concerning  the  expected 
frequency  of  detected  flaws.  Also,  management  oversight  and  level  of  accountability  also 
facilitate  reliable  inspection  programs.  Contributing  factors  play  a  key  role  as  well  in 
determining  the  sensitivity  of  a  procedure  to  human  factor  variations.  In  particular,  the 
design  of  NDE  equipment,  calibration  samples,  and  the  NDE  procedures  in  terms  of  level 
of  complexity,  usability,  environmental  conditions,  and  potential  for  interpretation  will 
contribute  to  the  variability  in  performance  for  different  operators. 

Representing  Human  Factors  in  Probabilistic  Models 

The  VNDE  software  platform  [6,7]  is  used  to  explore  the  effect  of  changes  to  the  NDE 
probability  of  detection  (POD)  model  related  to  human  performance  on  probability  of 
failure  and  total  cost  estimates.  A  four  parameter  POD  model  is  defined  as  follows: 


POD(a)  =  a  +  (j3  -  a)\  1  +  exp 


n 


In  a  -  In  a 


50 


,  -1 


(1) 


where  a  corresponds  to  the  flaw  size,  a  so  (a50)  corresponds  to  the  median  (50%)  detectable 
flaw  size,  c  represents  the  skewness  in  the  central  slope  of  the  POD  curve,  a  corresponds 
to  the  false  call  (FC)  rate  and  /?  is  defined  as  1  minus  the  random  miss  call  (RMC)  rate 
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[11].  Figure  1  presents  a  diagram  of  this  model  highlighting  these  four  parameters.  The 
real  challenge  in  understanding  the  sensitivity  of  human  factors  on  inspection  reliability 
and  maintenance  cost  concerns  representing  their  influence  on  each  of  the  four  parameters. 
False  call  rates  can  be  generally  attributed  to  a  lack  of  recent  experience  and  training  with 
the  procedure,  poor  concentration  during  setup  and  data  acquisition,  and  over-sensitivity  to 
image  or  signal  noise.  Random  missed  calls  can  be  attributed  to  several  factors:  poor 
integrity  of  the  operator,  lack  of  persistence  in  performing  the  procedure  under  all 
conditions,  and  a  false  expectation  of  the  frequency  that  cracks  will  be  found.  In  addition, 
differences  in  the  sensitivity  of  inspectors  to  visual  features  associated  with  flaws,  often 
associated  with  varying  levels  of  training  and  skill,  will  result  in  differences  in  the  median 
(50%)  detected  crack  length  parameter.  Likewise,  some  inspectors  will  be  better  skilled  at 
discriminating  between  subtle  flaw  signals  in  varying  levels  of  noise,  resulting  in 
differences  in  the  slope  (skewness,  a)  of  their  associated  POD  curves.  The  formulation  of 
quantitative  models  for  human  factors  for  varying  procedure  and  environmental  conditions 
is  a  long-term  need  and  a  significant  challenge,  but  would  benefit  this  evaluation  process. 

Software  for  Parametric  Design  Studies 

Prior  work  has  addressed  development  of  a  software  platform  to  enable  analysis  of 
tradeoffs  in  NDE  and  SHM  design  in  terms  of  product  life  cycle  outcomes  [6-7].  Several 
new  features  have  recently  been  incorporated  in  the  software  platform  to  facilitate 
specialized  design  studies.  Any  model  factor  can  now  be  selected  and  defined  as  a  variable 
for  a  parametric  study.  In  particular,  the  inspection  interval  can  be  varied  both  in  length  of 
time  for  each  in-service  period  and  number  of  total  inspection  intervals.  Also,  new 
visualization  and  tabular  features  have  been  included  in  the  software  to  explore  the  design 
space  in  terms  of  key  measures,  total  cost  and  maximum  probability  of  failure. 

Case  Study 

A  full  factorial  parametric  study  was  performed  varying  each  of  parameters  in  the  four 
parameter  POD  model  between  two  levels:  a  baseline  and  a  degraded  performance  level 
(Table  1).  The  hypothetical  maintenance  case  study  was  designed  including  three  in- 
service  intervals  with  two  inspection  periods.  The  POD  model  was  held  constant  for  the 
two  inspection  periods.  The  probability  of  failure  (POF)  model  included  both  probability 
of  fracture  and  probability  of  a  crack  growing  to  critical.  The  critical  flaw  size  was  0.75". 

The  results  from  the  parametric  study  are  displayed  as  a  function  of  total  life-cycle  cost 
and  maximum  probability  of  failure  in  Figure  2.  Each  of  the  sixteen  results  are  identified 
by  labels  indicated  the  factor  level  varying  between  baseline  (0)  and  degraded  (1) 
conditions  for  each  of  the  four  parameters  (a50,a,FC,RMC).  Trends  for  each  parameter  are 
also  presented  using  vectors  in  the  plot.  Degraded  performance  in  terms  of  increased 


FIGURE  1.  Four  parameter  POD  model. 
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TABLE  1.  Baseline  and  degraded  performance  levels  for  parametric  study. 


Parameter 

Baseline  Level  (0) 

Degraded  Level  (1) 

Units 

a50 

0.060 

0.070 

(in.) 

a  (skewness) 

0.02 

0.2 

0 

FC 

0.0 

1.0 

(%) 

RMC 

0.0 

10.0 

(%) 

FIGURE  2.  Plot  of  total  cost  versus  probability  of  failure  for  varying  the  four  POD  model  parameters 
(a50,a,FC,RMC)  between  two  levels,  highlighting  the  main  trends  with  respect  to  the  baseline  case  (0,0, 0,0). 


median  detected  crack  lengths  by  0.010"  ( associated  with  level  of  procedure  experience  of 
the  operator)  was  found  to  result  in  a  higher  POF  and  lower  total  cost.  A  reduction  in  the 
skewness  or  slope  of  the  POD  curve  (< associated  with  differences  in  skill  and  experience ) 
resulted  in  only  vary  small  increases  in  both  POF  and  total  cost.  An  increase  in  the  false 
call  rate  of  1%  (< associated  with  degraded  concentration  and  over-sensitivity  to  noise)  was 
found  to  increase  total  cost  with  little  impact  on  POF.  Lastly,  an  increase  in  the  percentage 
of  random  missed  calls  by  10%  {associated  with  a  lack  of  integrity,  poor  focus,  or  a  bias 
concerning  the  expected  frequency  of  detected  cracks)  was  found  to  significantly  increase 
the  probability  of  failure  with  a  slight  decrease  in  total  cost. 

Although  the  random  missed  call  rate  was  found  to  significantly  impact  the  probability 
of  failure,  the  degree  of  sensitivity  was  found  to  be  less  than  expected.  There  are  several 
reasons  for  the  probability  of  failure  results  not  being  more  significant  for  an  increase  in 
the  random  missed  flaw  rate  from  0%  to  10%.  The  first  source  concerns  the  nature  of  the 
POF  calculation.  Although  consisting  of  both  probability  of  fracture  and  probability  of  a 
crack  growing  to  critical,  the  POF  function  is  often  dominated  by  the  probability  of  fracture 
component.  Since  probability  of  fracture  is  dependent  upon  the  relation  between  stress 
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intensity  factor  and  crack  size,  which  is  quite  sensitive  to  changes  in  flaw  distribution  for 
very  small  crack  lengths  but  less  for  cracks  in  the  mid-range,  the  function  is  generally  less 
sensitive  to  changes  in  the  random  missed  call  rate  (RMC)  and  more  to  the  median 
detectable  flaw  size  (a50).  If  the  probability  of  failure  calculation  was  solely  dependent 
upon  a  crack  growing  to  the  critical  flaw  size,  changes  to  the  random  missed  call  rate  were 
found  to  be  much  more  significant  on  POF  with  respect  to  similar  changes  the  median 
detectable  flaw  size.  Second,  multiple  inspection  opportunities  also  significantly  reduce 
the  probability  of  a  missed  flaw.  In  particular,  if  inspections  are  independent  and  intervals 
between  inspections  are  short,  multiple  inspections  can  be  used  to  mitigate  poor  inspection 
performance.  Finally,  these  observed  changes  should  not  be  considered  to  be  absolute 
trends,  but  are  dependent  on  the  equivalent  initial  flaw  size  distribution,  flaw  growth  model 
and  interval  of  inspection.  Through  these  simulated  studies,  unexpected  insight  was 
achieved  concerning  the  influence  of  the  POD  parameters  on  cost  and  reliability. 

SENSOR  DEGRADATION  IN  STRUCTURAL  HEALTH  MONITORING 

Understanding  the  likelihood  and  impact  of  degradation  on  an  in-situ  sensor  network 
for  structural  health  monitoring  is  critical  for  deployment  of  these  systems.  Two  classes  of 
SHM  systems  are  considered  here:  global  health  monitoring  systems  incorporating 
distributed  sensors  such  as  strain  gauges  and  acoustic  emission  transducers,  and  local 
methods  for  critical  structural  locations  using  ultrasonic  and  eddy  current  sensors. 
Evidence  from  existing  strain  gauge  sensor  data  on  C-17  aircraft  demonstrated  this  issue, 
where  22%  of  the  sensors  were  infant  failures  and  about  40%  of  the  total  failed  within  the 
first  ten  years  of  the  aircraft  life  [12].  This  likelihood  for  a  significant  percentage  of 
sensors  to  fail  during  the  service-life  of  an  aircraft  mandates  sensor  redundancy  for  global 
SHM  methods  and  sensor  self-monitoring  and  maintenance  for  local  SHM  methods. 
Certification  requirements  for  in-situ  sensors  are  being  prescribed  for  military  and 
commercial  SHM  applications  [13].  In  addition  to  standard  certification  tests  for  electrical 
components,  in-situ  sensors  must  also  demonstrate  their  reliability  to  detect  a  range  of 
expected  flaw  conditions  over  the  total  expected  life  of  the  structure  and  beyond. 
However,  NDE  sensors  do  not  maintain  a  consistent  sensitivity  and  require  frequent 
recalibration.  Also,  degradation  of  the  sensor  bond  through  thermal  and  dynamic  loading 
has  been  recently  demonstrated  and  must  be  addressed  [14].  Self-calibration  methods  have 
been  proposed  to  help  address  some  sensor  and  bond  variation  over  time  [15,16]. 
However,  a  better  understanding  of  the  cost-benefit  of  maintaining  in-situ  sensor  networks 
for  an  aircraft  service  life  is  needed.  In  this  paper,  probabilistic  models  are  presented  to 
study  the  impact  of  SHM  degradation  and  maintenance  on  reliability  and  cost. 

Probabilistic  Model  for  SHM  Degradation 

A  methodology  has  previously  been  presented  for  incorporating  SHM  systems  into  a 
design  platform  for  enabling  analysis  and  optimization  of  tradeoffs  in  terms  of  reliability, 
cost,  and  availability  [7],  A  summary  of  the  SHM  probabilistic  model  and  new  features 
addressing  sensor  degradation  are  presented.  From  the  perspective  of  quantifying  the 
reliability  of  a  SHM  system,  there  is  an  underlying  relationship  that  must  be  evaluated 
between  accuracy  in  the  damage  state  estimate  (d)  with  respect  to  the  actual  damage  state 
(a),  with  special  interest  placed  on  the  critical  flaw  size  ( acr )  that  prompts  a  maintenance 
action.  As  with  NDE  procedures,  the  relationship  between  the  flaw  size  and  the  probability 
of  detection  and  false  call  rate  for  the  SHM  system  is  directly  represented  using  a  four 
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parameter  probability  of  detection  model.  To  represent  degradation  over  time,  the  four 
parameters  of  the  probability  of  detection  curve  can  be  defined  as  a  function  of  time,  where 


n  flna-  ju(t ) 

S  l  cr (t) 


-1 


POD(a, t )  =  a(t )  +  {Pit)  -  a(f)  k  1  +  exp 


(2) 


Figure  3  presents  a  plot  of  a  POD  function  that  varies  with  time,  representing  potential 
degradation  of  the  SHM  process  through  changes  in  the  50%  detectable  flaw  size  and  the 
random  missed  flaw  rate.  Time  dependent  models  should  include  early  (infant)  failures 
occurring  over  short  periods  and  slow  degradation  changes  in  time  of  the  sensor  system. 

The  costs  associated  with  the  degradation  of  structural  health  monitoring  systems  can 
be  categorized  as  development  costs,  implementation  costs,  and  in-service  costs. 
Additional  development  costs  are  necessary  to  design  in-situ  sensors  with  self-diagnostics 
capability  and  optimize  sensor  redundancy  in  case  of  sensor  failure.  Implementation  costs 
will  include  additional  validation  costs  to  ensure  the  sensor  system  will  maintain  acceptable 
reliability  over  the  life  of  the  aircraft.  In-service  costs  include  the  additional  cost  of  fuel 
due  to  added  SHM  system  weight,  data  interpretation  labor  costs,  SHM  maintenance  costs 
associated  with  failed  and  degrading  sensors,  and  the  cost  of  secondary  inspection  and 
unnecessary  repair  due  to  false  calls  or  unnecessary  calls  when  flaws  are  very  small.  The 
percentage  of  secondary  inspections  and  repairs  can  be  evaluated  through  probabilistic 
models  and  used  as  inputs  into  the  cost  model.  Probability  of  failure  models  for  sensors, 
in  particular  for  local  sensors  requiring  immediate  replacement  upon  failure  to  maintain 
damage  observability,  may  also  be  included  to  supplement  the  maintenance  cost  model. 

Case  Study 

A  case  study  explores  degradation  in  SHM  system  performance  over  time.  The  total 
service  life  was  divided  into  ten  service  periods  separated  by  nine  maintenance  events 
consisting  of  SHM  data  processing  and  subsequent  field  inspection  and  repair.  A  variable 
probability  of  detection  function  was  assigned  to  the  SHM  system  as  follows:  maintenance 
events  1-5  were  assigned  the  SHM  POD  labeled  to  in  Figure  4(a),  tj  was  assigned  to  the 
SHM  system  at  maintenance  event  6,  0  to  maintenance  event  7,  and  0  to  maintenance 
events  8  and  9.  Only  slow  sensor  degradation  is  considered  in  this  hypothetical  case  study. 

Figure  4(b)  compares  the  probability  of  failure  history  for  the  time  varying  SHM  POD 
case  just  described  to  that  of  a  case  where  the  POD  of  the  SHM  system  is  fixed  to  that 
labeled  to  in  Figure  4(a).  Figure  4(b)  shows  that  the  variable  POD  case  results  in  an 
undesirable  increase  in  probability  of  failure.  If  such  a  condition  is  experienced  in  the 
field,  replacement  of  the  sensors  to  maintain  inspection  reliability  is  suggested.  Not  shown 
here  is  the  fact  that  the  total  cost  for  the  variable  POD  case  is  lower  than  that  of  the  fixed 


POD(a.t) 


a 


FIGURE  3.  Probability  of  detection  (POD)  function  with  variability  in  model  parameters  over  time. 
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Time  Varying  Probability  of  Detection  Curve 


FIGURE  4.  Effects  of  SHM  system  deterioration:  (a)  time  varying  POD,  (b)  resulting  probability  of 
failure. 


POD  case,  because  finding  and  repairing  less  flaws  results  in  lower  costs  at  the  expense  of 
a  higher  risk  of  failure.  However,  if  degraded  performance  leads  to  increased  false  calls, 
the  model  indicates  that  much  of  the  earliest  calls  will  be  either  false  or  premature  calls  of 
very  small  cracks.  Procedures  must  be  in  place  to  manage  the  occurrence  of  such  false 
calls  early  in  the  implementation  of  SHM  systems  through  secondary  expert  review  of  data 
and  inexpensive  follow-up  inspections  to  mitigate  unexpected  costs  and  issues  concerning 
trust  in  the  system  with  the  aircraft  maintainer. 

CONCLUSIONS  AND  RECOMMENDATIONS 

To  perform  a  quantitative  evaluation  of  the  influence  of  human  factors  on  NDE  and 
sensor  degradation  on  SHM  applications,  probabilistic  models  were  developed. 
Quantitative  probabilistic  risk  assessments  and  cost  evaluations  were  presented  concerning 
the  effects  of  variations  in  probability  of  detection  associated  with  human  factors.  In 
addition,  in-situ  sensor  degradation  on  life  cycle  measures  was  studied  in  terms  of  cost  and 
probability  of  failure.  Through  simulated  studies,  insight  was  presented  concerning 
possible  opportunities  and  pitfalls  of  SHM  applications.  Design  tools  have  also  been 
integrated  in  the  software  platform  to  easily  explore  a  variable  number  of  inspection 
intervals  and  variable  length  of  each  service  period. 

Future  work  will  explore  the  sensitivity  of  model  trends  to  cost  parameter  levels,  and 
acquire  better  data  on  flaw  size  distributions  and  real  costs  (of  repairs)  for  promising 
applications.  Long  term  efforts  plan  to  explore  the  development  of  numerical  and 
empirical  models  to  better  represent  variations  in  human  factors  in  NDE  models  and  sensor 
degradation  in  SHM  models.  Probabilistic  models  will  be  studied  to  address  the 
assumption  of  statistical  independence  for  multiple  measurements  of  the  same  component 
or  inspection  system  over  time.  Lastly,  to  best  address  the  management  of  the  vast  array  of 
critical  structural  locations  over  the  service  life  of  an  aircraft  fleet,  a  ‘hybrid  approach’  to 
fleet  management  is  encouraged  considering  a  case-by-case  evaluation  of  the  most 
appropriate  maintenance  approach:  1)  fail-safe  design  (no  inspection),  2)  scheduled 
nondestructive  inspection,  3)  loading  condition  monitoring,  4)  damage  state  monitoring,  5) 
load  condition  monitoring  with  condition-based  maintenance,  and  6)  damage  state 
monitoring  with  secondary  nondestructive  inspection. 
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